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ABSTRACT 



This study investigated the use of tools and design features 
as employed in a problem-based learning (PBL) environment and their 
effectiveness on seventh grade students' learning of science concepts. 
Specifically, the study examined: (1) the effect of the computer- supported 

PBL environment on the achievement of middle school students; (2) the effect 
of the computer-based PBL environment on middle school students 1 attitudes 
toward science; and (3) the relationship between students’ math or reading 
ability and their achievement in the PBL environment. Dependent measures were 
achievement, degree and frequency of access to the tools/features, and 
attitude toward learning science. The treatment consisted of three groups: 
computer- supported PBL environment, paper-based PBL, and a control group. 
Results indicated that: both the groups that used the computer- supported and 
paper-based PBL significantly improved their achievement scores, while the 
increase for the control group was not significant; there was no significant 
difference between the computer -supported and paper groups; students' reading 
ability is a better predictor for students' achievement in a PBL environment 
than their math ability; and students' attitude toward learning science was 
not affected by the introduction of PBL. Two tables and a graph present data. 
Contains 14 references. (DLS) 
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Abstract: A nagging problem concerning school-based learning is the difficulty many PERM,SS, 0 N T 0 REPRODUCE THIS 

students expenence in applying what they have learned to everyday situations. Problem- 

based learning (PBL) emphasizes solving authentic problems in authentic contexts and has G,H, Marks 

been shown to be an effective instructional method. Though literature supports the efficacy 

of problem-based learning and the benefits of problem-based learning are obvious, little 

research exists which investigates the types of tools or features that are effective in 

supporting students working in PBL environments. This study examines how middle school TO THE EDUCATIONAL RESOURCES 
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1. Research Framework 

Historically, schools have attempted to teach the content and skills essential for individuals to 
function in society. Since the Industrial Revolution schools have emphasized the learning of basic skills. Yet 
today it is clear that this type learning is not enough. More jobs than ever require problem solving and critical 
thinking. Dewey [Dewey 1938] argued that intelligence is the ability to clarify a goal and develop a plan for 
its attainment. Authentic learning environments such as problem-based learning embody this in its problem 
definition and resolution. 

With the shift from a behavioral to a cognitive perspective of learning, researchers have become 
increasingly interested in cognitive processes and their effect on the construction of knowledge [Driscoll 
1994]. Many cognitive psychologists and constructivists believe that learning cannot be separated from the 
learning activity and context [Bednar, Cunningham, Duffy, & Perry 1992]. 

A nagging problem concerning school-based learning is the difficulty many students experience in 
applying what they have learned to everyday situations. Many schools are concerned with the transfer of 
abstract, decontextualized concepts where knowledge is viewed as a discrete object residing outside of the 
individual [Brown, Collins, & Duguid 1989]. Increasing amounts of research indicate that the inability of 
students to apply concepts learned in formal contexts is due to the abstraction and decontextualization of 
learning [Cognition and Technology Group at Vanderbilt 1992]. Spiro and Jehng [Spiro & Jehng 1990) argue 
that the decontextualization of knowledge creates inflexible understandings. But it is not the abstraction of 
knowledge as such that distracts learners, but that the abstractions are not illuminated with examples in 
context. Understanding is a product of the context and activity [McLellan 1993]. 

Context provides a framework which guides and supports the learner. The context naturally 
structures knowledge in a way that suggests its proper use. Traditional instructional design attempts to 
simplify learning by dividing the content into components and teaching them separately. Situated cognition 
argues that learning is simplified by embedding concepts in the context in which they will be used [Brown & 







Duguid 1993]. The context provides the much needed support when working at the edge of one’s ability. Yet 
authentic context alone is not enough to support student learning. Situated cognition argues that learners must 
engage in authentic tasks as well [Winn 1993]. Authentic activities are the ordinary practices of the experts 
in that domain [Brown et al. 1989]. The learning activity is an integral part of what is learned. An activity in 
which all domains engage is the solving of problems. Experts utilize tools to support problem-solving in 
context. Active use of a culture’s tools provide learners the opportunity to view knowledge from the experts’ 
perspective [Brown et al. 1989]. Active learning in the context of a domain’s culture allows the gradual 
“fleshing out” of concepts over time. Each use of a concept or tool will further develop the learners’ 
understanding. 

There has been a growing body of research on authentic and situated learning environments utilizing 
the problem-based approach to learning [Spiro & Jehng 1990]. Problem-based learning (PBL) emphasizes 
solving authentic problems in authentic contexts. It is an approach where students are given a problem, 
replete with all the complexities typically found in real world situations, and work collaboratively to develop 
a solution. Problem-based learning provides students an opportunity to develop skills in problem definition 
and problem solving, to reflect on their own learning, and develop a deep understanding of the content 
domain [Lajoie 1993] [Jacobson & Spiro 1995). This approach was developed in the fifties for medical 
education, and has since been used in various subject areas such as business, law, education, architecture and 
engineering. Most recently, there is a growing interest among educators to use problem-based learning in the 
K-12 setting, and a growing need for problem-based educational software to facilitate the development of 
higher order thinking skills via technology. 

Though literature supports the efficacy of problem-based learning and the benefits of problem-based 
learning are obvious, little research exists which investigates the types of tools or features that are effective in 
supporting students working in PBL environments. In order to design an effective computer-supported PBL 
learning environment, it is important to understand the tools and design features included in the software, and 
their impact on learning. It is, therefore, the purpose of this study to examine and understand how middle 
school students use and interact with a recently available computer-supported PBL environment developed by 
a major publishing company. 



2. Research Questions 

This study investigates the use of tools and design features as employed in a problem-based learning 
environment and their effectiveness on middle school students’ learning of science concepts. Specifically, the 
study asked the following questions: (1) What is the effect of the computer-supported problem-based learning 
environment on the achievement of middle school science students?, (2) What is the effect of the computer 
supported problem-based learning environment on the middle school students’ attitudes toward science?, (3) 
Is there a relationship between students’ math or reading ability and their achievement in the problem-based 
learning environment? 



3- Design of Study 

3.1 Sample 

The participants of the study (N = 115) were students enrolled in seventh grade science at a middle 
school located in a medium-sized city in the southwestern United States. The school has a high percentage of 
minority students. The participants in the study consisted of 66% Hispanic Americans (N=76), 12% African 
Americans (N=14) and 22% white (N=25). The age of the students ranged from 12 - 14 years. Of the 
participants in the study, 50 were male and 65 female. 

3.2 Treatment 
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The treatment consisted of three groups: computer- supported PBL environment, paper-based PBL, 
and a control group. Students from three intact classes were randomly assigned to each of the conditions 
described below. Because the treatment was included as a part of the regular science classes, complete 
random assignment was not possible in this case. There were 59 students in the computer-supported PBL 
environment, 38 in the paper-based PBL, and 18 in the control group. 



3,2.1 Computer-Supported PBL 

The computer-supported PBL group used problem-based learning software recently developed by 
Holt, Rinehart and Winston Publishing Company. The CD-ROM-based program contains eight activities on 
different topics developed to support the middle school science curriculum. Upon starting the PBL 
application the students find themselves in a virtual science laboratory. Immediately on entering the 
laboratory a short video segment plays in which a scientist provides important details about a scientific 
problem she is working on and solicits the students’ help in developing a solution. The activity used in this 
study is concerned with the classification of a microorganism and was selected because it was related to the 
topic being studied in the classes participating. 

The students are supported in finding a solution to the problem by the availability of tools in the 
virtual laboratory. An expert scientist built into the virtual laboratory provides information about the scenario 
and takes on the role of a mentor in which she provides hints and feedback during the activity. The inbox 
provides a printed version of the problem scenario thereby supporting students in defining the problem. A lab 
manual provides important information on the use and importance of the tools found in the lab. The lab also 
furnishes a computer database which contains information that is essential for solving the problem. A 
notepad is also provided to support students in note taking during the activity. Most students are able to 
complete the activity in approximately forty-five minutes. 



3.2.2 Paper-based PBL 

Other students worked through a problem-solving activity, equivalent in content and available 
resources to the computer-based software, except that the information given (including the information in the 
database and the pictures) was print-based. Though identical in the resources provided, the computer and the 
paper-based PBL are different in that (1) the computer-based version allows interactive access of information 
while the paper-based does not, and (2) the computer-based version provides information in multimedia 
format while the paper-based does not. 



3.2.3 Control Group 

A third class was included as a control group. Students in this class learned the same content on 
microorganisms using the traditional approach (mostly lecture-based). 



3.3 Dependent Measures 

There are three dependent measures in the study: (1) achievement, (2) degree and frequency of 
access to the tools/features, and (3) attitude toward learning science. Students’ knowledge about 
microorganisms was assessed through a pretest and a posttest. Their use of the tools and various design 
features in the computer version was recorded. Their attitudes toward science were evaluated through a 
questionnaire entitled “Attitude Toward Science in School Assessment.” This attitude questionnaire addresses 
students’ feelings about science as a subject and consists of 14 Likert scale items, with a reported reliability 
of .95 [Germann 1988]. In addition, the students in both the computer and paper groups were asked to record 
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their steps in solving the problem. Interviews with students from all three groups and with the teacher were 
conducted. The triangulation of the quantitative and qualitative data sources will help to answer the research 
questions with richer and more detailed information. 



3.4 Procedures and Analyses 

Each of the 5 classes involved in this study was assigned randomly to one of the treatment 
conditions. For students in the computer and paper PBL groups, the entire treatment consisted of two phases. 
First, the teacher modeled the problem-solving process using another activity from the computer software (for 
the paper group, the information was presented through paper and pencil). Then the students were divided 
into groups of four or five and asked to work through the activity on microorganisms collaboratively. 
Literature on PBL shows that providing necessary scaffolding is a critical step in making PBL successful. 
The purpose of this teacher modeling phase was to introduce students to the steps of problem solving and 
provide them with guided practice. This phase was included in both the computer-supported and paper-based 
environments because we felt that the modeling should be part of problem solving regardless of the media 
involved and our primary interest in this study was to find out if the hypermedia environment could provide 
additional support for the students. Therefore, the instruction of the two conditions were held to be same. 
The control group worked through the regular instruction on microorganisms with no intervention. 

In order to answer the first research question, “What is the effect of the computer-supported 
problem-based learning environment on the achievement of middle school science students?,” a two-factor 
mixed ANOVA was run with the grouping (computer, paper and control) as a between-subjects independent 
variable, and the data collection points (pre vs. post) as the repeated measure independent variable. The 
dependent variable was the pre and post achievement scores in the science content test. 

In order to answer the second research question, “What is the effect of the computer supported 
problem-based learning environment on the middle school students’ attitudes toward science?,” a two-factor 
mixed ANOVA was run with the grouping (computer, paper and control) as a between-subjects independent 
variable, and the data collection points (pre vs. post) as the repeated measure independent variable. The 
dependent variable was the pre and post scores for the attitude questionnaire. 

In order to answer the third research question, “Is there a relationship between students’ math or 
reading ability to their achievement in the problem-based learning environment?,” a multiple regression was 
run with students’ math and reading ability, measured by their most recent Texas Assessment of Academic 
Skills (TAAS) as the independent variables and their achievement test as the dependent variable. 

In addition, students’ use of tools in the hypermedia environment and the problem solving steps 
students taken in the computer and paper groups were analyzed descriptively. Students were selected for the 
post interviews after they completed the study. The purpose of the interviews was to find out (1) what the 
students liked and disliked about the environment; (2) what they found most useful in the environment; (3) 
what they thought about the process in general. Interviews were included as part of the data sources in order 
to provide more information on the study and substantiate the quantitative analyses. The interview data will 
be analyzed according to Miles and Huberman’s framework of qualitative data analysis [Miles & Huberman 
1994]. Such descriptive analyses and the findings from the qualitative data are being conducted at this point 
and will be added to the conference presentation. 



4. Results and Discussion 

The results of the two-factor mixed ANOVA on achievement indicated that there was a significant 
two-way interaction between the grouping (computer, paper and control) and the data collection points (pre 
vs. post) for the achievement scores: F (2, 96) =5.50, p <.01. All groups increased their achievement scores 
from pre to post. The gains from pre to post were significantly greater for the computer and paper groups 
than for the control group [Tab. 1] [Fig. 1]. The gain differences between the computer and the control 
groups, and the paper and the control groups were significant at p < .05 level based upon the Fisher’ s PLSD 
post hoc tests. The gain difference between the paper and the control groups was also significant at p < .05 
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level for the Scheffe post hoc test. This finding shows that both the computer and paper groups have 
significantly improved their achievement scores after they participated in the study while the increase for the 
control group was not significant. In other words, there was an effect of the problem-based learning 
environment on the achievement of middle school science students. Yet the difference between the 
computer-supported PBL and paper PBL was not significant. 

The results of the two-factor mixed ANOVA on attitude indicated that there was not a significant 
two-way interaction between the grouping (computer, paper and control) and the data collection points (pre 
vs. post) for the achievement scores: F (2, 95) =.26, p= .77. Although there was a small increase in attitude 
from pre to post for all groups, there were no differences among the three groups [Tab. 1]. In other words, the 
treatment did not have an impact on the students’ attitudes toward learning science. 
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N* 
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Post 
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Pre 


Post 


Computer Group 


50 


12.7 (12.52) 


31.58 (20.7)** 


48 


28.23 (12.19) 


30.00(14.10) 


Paper Group 


32 


8.22(12.53) 


28.38 (18.43)** 


17 


31.91 (9.98) 


34.76(11.34) 


Control Group 


_LZ_ 


27.35 (12.64) 


32.47 113.351 


23_ 


25.82 18.901 


29.82 114.331 



* N indicates the number of students who turned in both the pre and post evaluation forms for each group. 

** significantly different from the control group, p < .05 

Table 1: Mean and Standard Deviation (in Parenthesis) on achievement and attitude for the Computer, the 
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3.46, p < .1 
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1.26, p =.21 


intercept 


4.13 
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Beta Weight 





reading ability .42 reading ability .47 

math ability J_2 math ability JJZ. 




Table 2: Results of the Multiple Regression Analysis 

We were interested in finding out if there existed a relationship between students’ math and reading 
abilities and their achievement when working in PBL environments. Students’ scores in reading and math 
from a most recent TAAS test were used. TAAS is a state-wide testing system that assesses the overall 



academic achievement of all students in Texas at different grade levels. It provides information on students’ 
reading ability, math ability and writing ability. The results of the multiple regression analysis showed that 
the significance of the relationship was moderately high among reading ability, math ability and the 
achievement scores for students using PBL: r =.59, p < .01. This significant relationship was mainly attributed 
to students’ reading ability t(12) = 3.46, p <.01 [Tab. 2]. That is, students’ reading ability is a better predictor 
for students’ achievement in a PBL environment than their math ability. A PBL environment relies much on 
problem identification, presentation, problem-solving and student reflection. Though mathematical ability is 
obviously very important in problem solving, being able to read and comprehend the problem is critical. This 
finding provides some evidence on this issue and suggests that in order for students to be successful in a PBL 
environment, we, as teachers, need to make greater efforts to increase students’ reading levels. 

The findings of this study suggest that problem-based learning can influence the middle school 
students in their learning of science. When students were exposed to the PBL environment, they increased 
their achievement scores more than those students who learned the same content in the traditional classroom. 
Consistent to the literature on PBL, the results of the study provide some evidence in supporting the use of 
PBL. The findings also suggest that both the computer-supported and paper based PBL are equally effective. 
An interest of this study was to find out the types of tools or design features that can support students when 
working in a computer supported PBL environment. Although there was no significant difference between 
the computer and the paper groups on achievement, initial examinations of the problem solving process by the 
computer and the paper groups show some differences in the steps and use of available resources/tools. The 
use of tools/resources are currently being analyzed. The findings of the analyses for tool use and the 
qualitative data will be added to the result and discussion section for the conference presentation. Together, 
the findings will provide more insights as to how the tools can or cannot support problem solving in 
computer-supported PBL environments. 

No significant differences among the three groups were found on the middle school students’ attitude 
toward learning science. That is, the middle school students’ attitude toward learning science was not 
affected by the introduction of PBL, computer-supported or paper-based. 
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